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ABSTRACT 

Training interventions can be evaluated by calculating return 
on investment (ROI) and cost-benefit analysis. The four-level model proposed 
by Kirkpatrick is the dominant evaluation model used. Calculating ROI has 
been a critical issue for trainers and executives, but only a few 
organizations have implemented the process that is considered as difficult, 
impractical, and expensive. Since organizations have moved from training for 
activity to training for impact, determining ROI is a key process to measure 
training accountability. Strengths of ROI are that organizations connected 
training with strategic goals, used a variety of methods to collect data and 
implemented attempts to isolate training effects, defined units of 
measurement, calculated monetary value of improvements, and gathered 
measurements before and after training. These successful practices have 
stated accountability for participants, managers, and HRD staff. Challenges 
are that each organization applies different methods to determine costs; 
practitioners are not implementing appropriate methods to isolate the 
influence of training on organizational goals; and results from HRD programs 
are not converted into monetary values. Major conclusions are that ROI 
implementation , has relevant benefits for organizations; isolation and value 
of money over time are two important elements that should be considered; and 
HRD practitioners would benefit from applying principles of research design 
to evaluation processes. (31 references) (YLB) 
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Abstract 

The field of human resource development (HRD) is related to training, education, and 
development interventions (Phillips, 1997a). This literature review will focus on evaluation of 
training interventions by calculating return on investment (ROI) and cost-benefit analysis (CBA). 
The first section will give definitions for key concepts in this paper, namely evaluation, ROI, and 
CBA. In addition, it will briefly include main current issues about training evaluation and the 
application of ROI and CBA to training interventions. Next, the study will synthesize relevant 
findings on both, strengths and challenges of some experiences where a variety of organizations 
have applied ROI and CBA for their training programs. Finally, the third section will present 
conclusions and implications of these findings. 

Keywords: Evaluation Methods; Training; Training Effectiveness; Performance Improvement; 
Organizational Training; Professional Development; Program Improvement. 
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Evaluating Training: Return on Investment and Cost-Benefit Analysis 

Key concepts will be considered throughout this literature review, namely evaluation, 
return on investment, and cost-benefit analysis. Evaluation is currently one of the most crucial 
issues faced by HRD practitioners is evaluation of HRD interventions (Holton, 1996). HRD 
departments can no longer continue using organization’s resources without proving their 
contribution to its strategic goals. Consequently, issues about evaluation processes, models and 
methods are relevant for most organizations (Phillips, 1997b). That is why it is relevant to define 
evaluation. According with Kaufman and Keller (1994), evaluation is a process of gathering 
information for decision-makers to compare the training program results with its intentions. 
Nonetheless, a holistic evaluation should also assess those methods and resources used to deliver 
results. In correspondence, Phillips (1997b) states that evaluation is a systematic process that 
gathers relevant information to determine the value or meaning of an activity or process. Thus, an 
evaluation process is implemented to make decisions about the future of a training program or to 
improve it. 

Another important key concept is Cost-Benefit Analysis (CBA). This type of analysis is 
based on benefit/cost ratio (BCR). This ratio divides total benefits by costs. For analyses of 
training programs, it would be the result of dividing program benefits by program costs. The 
formula for this concept is the following one (Phillips, 1997b): 

BCR= Program Benefits / Program Costs 

Another related concept is Return on Investment (ROI). It is a percentage that expresses a 
relationship between monetary net-benefits and costs of an investment. Net benefits are 
calculated by subtracting costs from the total benefits, which are then divided by total costs. 
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Then, 100 multiply this result. The formula for this concept is the following one (Phillips, 

1997b): 

ROI(%)= [Net Program Benefits / Program CostsjxlOO 
Evaluation of Training Programs 

Recently, several authors have confirmed that training efforts are big business in America 
based on recent reports about the amounts invested in this area. For example, in 1995, employers 
spent $55.3 billion on training (Brauchle, & Koenecke, 1999; Flynn, 1998). These conditions 
have made managers inquiry whether or not training is paying off the investment. That is one of 
the reasons why organizations are switching from training for activity toward training for impact 
paradigm. This new paradigm creates a partnership among key actors associated with training 
programs. Furthermore, HRD staff identifies business needs that are driving the request for 
training. Finally, a requisite in this new paradigm is measurement of training results (Robinson, 

& Robinson, 1989). That is why in the last twenty years, evaluation of training interventions have 
become a significant topic of concern for trainers, managers, and researchers in United States and 
Canada (Dionne, 1996). 

While trainers have focused on how training affects individuals, managers are concerned 
with organization’s overall performance. Therefore, managers consider that it is not enough to 
assume that if training affected the individual, he/she will improve his/her personal performance 
and then the organization’s performance will improve too (Lyau & Pucel, 1995; Whalen, 1997). 

HRD practitioners have applied different models to evaluate training programs. Currently, 
the four-level model proposed by Kirkpatrick is the dominant evaluation model used and known 
by HRD practitioners (Holton, 1996; Lyau & Pucel, 1995). This model proposes four levels, 
namely reaction, learning, transfer and results. The first level assesses participants’ reactions to 
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the training experience. The second level assesses participants’ learning experience, that is 
whether or not they acquire the knowledge, skills, or attitudes needed. The third level focuses on 
determining whether or not those skills are being transferred on the job. Finally, the fourth level 
evaluates how the training program has contributed to improve organization’s expected results 
(Kirkpatrick, 1994). 

Even though the four-level model is the most common among practitioners, some 
authors found that the four-level model does not always provide the type of evidence required by 
managers (Holton, 1996; Lewis, 1996; Lyau, & Pucel, 1995). Therefore, several authors have 
gone beyond by proposing some changes to the model. For example, Kaufman (1994) proposed 
adding a new level to this method, namely evaluation of social and community impact of HRD 
interventions. Phillips (1994) suggested the implementation of ROI process as a fifth level for the 
four levels of evaluation as defined by Kirkpatrick. Some other authors suggested that the 
evaluation process can be implemented even before starting a program, that is to forecast whether 
or not a program is worthy to implement (Swanson & Gradous, 1988). 

Current Issues on Implementing ROI and CBA to Evaluate Training 

As economic entities, if organizations want to survive they should judge each of its 
components from a return on investment (ROI) framework. Under these criteria, there are four 
perspectives of HRD function in the organization. One view considers HRD as a major business 
process, something an organization must do to succeed. The second one states HRD a value- 
added activity that is potentially worth doing. The third view perceives HRD as an optional 
activity, something that is nice to do. Finally the last one conceives HRD as a waste of business 
resources, something that means more costs than benefits for the organization. The dominant 
views are the two last ones (Swanson, 1998). 
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Although some HRD practitioners feel that evaluating training in financial terms is 
necessary for advancing its role as an important business partner, others point out that HRD has 
long been relied upon to meet a variety of developmental needs throughout the organization, 
whether or not its benefits can be expressed in financial terms (Blickstein, 1996; Torraco, 1995). 
Nonetheless, the language of business is dollars and financial measures are the most widely used 
measures of performance in business. Measuring ROI is shown to be a technique that 
demonstrates the power of HRD to advance the organization’s competitive position in a manner 
that is credible to decision makers throughout the organization Therefore, HRD programs can be 
measured and analyzed with ROI, and other financial analysis and measures. Financial analysis is 
crucial for building HRD’s image as contributors in the organization value chain (Torraco, 1995). 

Besides those internal factors, some external conditions, namely high levels of 
competition all around the world, are pushing organizations to prove that their HRD department 
is contributing to the bottom-line (Holton, 1996). Consequently, management at both private and 
public organizations is questioning the return on investment in training (Brauchle & Koenecke, 
1999; Phillips, 1997a; Dionne, 1996; Lyau & Pucel, 1995). 

According with Phillips (1997a), even though calculating ROI has been a critical issue for 
both trainers, and executives, only few organizations have implemented the process. The 
American Society for Training and Development ( ASTD) reported that only three percent of 
HRD programs are evaluated for financial impact (Swanson, 1998). 

The reason is that ROI process is often considered as difficult, impractical, and 
expensive. ROI calculation has been easier to calculate for hard-data, such as productivity, 
quality, and time, than for soft-data, such as customer satisfaction, employee turnover, and job 
satisfaction (Brauchle, & Koenecke, 1999). That is why some opinions state that calculating ROI 
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could become only a seductive idea, because it is subjective and sometimes not worth the effort, 
especially for soft data. Flynn (1998) presented different cases when ROI calculation was 
appropriate. The author consider that ROI is pertinent to evaluate one-time training programs on 
specific skills (hard data), in which few other factors will affect the outcome. 

Accordingly, Brauchle and Koenecke (1999) affirm that for most organizations, ROI 
seems to be impractical and expensive to calculate for all of their training programs. However, 
Phillips (1996c) proposes some practical ways to convert program results into monetary benefits 
without increasing costs too much. Organizations planning to implement a ROI process level of 
evaluation need to gather data from all four levels. The reason is that HRD staff has to make sure 
that a chain through the four levels has occurred. Once the skills and knowledge are learned 
(level two) they are been applied on the job (level three) to produce business results (level four). 
These data are relevant to concluded whether or not is training the cause of the results been 
measured (Phillips, 1997b). 

In summary, the dilemma about the ROI process is a source of frustration with many 
senior executives and even within the HRD field itself Most executives realize that training is 
needed for organization’s growth and restructuring phases. Therefore, the same executives 
logically expect that training can pay off in important bottom-line measures, namely productivity 
improvements, quality enhancements, cost reductions, and timesavings. Organizations have 
moved from training for activity to training for impact (Robinson, & Robinson, 1989). In this 
new paradigm, determining ROI is a key process to measure accountability of training (Phillips, 
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generated more fees, got involved in future income-producing activities, and a significant return 
on investment. 

In summary, there are several strengths found in these experiences and research. First, 
organizations connected training with strategic goals; training is based on needs assessment, 
content is supported on job task analysis or validated by subject matter experts, and evaluation is 
considered at different levels of the program design (Phillips, 1996a). This is one of the most 
relevant strengths, because as Allen (1996) stated principles of good instructional design and 
development are being considered and should continue this way. For example, regarding 
computer-based training, these programs are generating profit and expected performance when 
they are founded on sound business plans and focused on audience needs. Second, organizations 
use a variety of methods to collect data, namely action plans, follow-up sessions, performance 
monitoring, management estimation, follow-ups with survey, observations, interviews, and 
questionnaires. Third, organizations implemented some attempts to isolate the effects of training, 
define units of measurement, calculate monetary value of improvements, and measures are 
gathered before and after training (Phillips, 1996a). Fourth, these successful practices have stated 
accountability for participants, managers, and HRD staff, as well. Fifth, calculation of ROI was 
based on estimations made by participants, supervisors, and/or managers. Sixth, time was 
allowed to design the program, implement the training process, and new skills development that 
could show results. Seventh, some organizations reported costs for several stages of the process. 
Finally, the training program had a support system, CEO was involved, and steering committees 
conducted the process. 
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Applications of ROI and CBA in Organizational Training 

Most of the literature on calculating ROI for training programs shows their successful 
experiences by sharing their ROI. For example, Cadence Design Systems determined that its 
sales-training program had a three-year cost of $1.4 million, and had a return on investment 
greater than 1,700% (Anonymous, 1997). Likewise, Stentor Resource Center determined the 
existence of relevant amount of savings as a result of measuring its learning program. Some key 
elements in the process were planning process focused on results, real-time learning on real 
projects, shared learning, ongoing evaluation process, and keep records of experience 
(Simington, 1998). However, none of these cases explain in detail how they came up with those 
successful results. 

Brauchle and Koenecke (1999) affirm that on one hand, some human resource managers 
are reluctant to share the applied processes to calculate ROI. On the other hand, there are few 
case studies that explain how the training department calculates ROI. 

The following findings are based on some research findings and 15 case studies in which 
training programs are evaluated demonstrating their return on investment. Jack Phillips published 
these cases in 1994 as a result of an effort of seeking for the best practices on implementing ROI. 
First, the strengths of these experiences and research are described. Second, relevant challenges 
are illustrated. 

Strengths of Successful Experiences on Calculating ROI and CBA for Training Programs 

Penske Truck leasing company developed a training program for supervisors. The 
organization started with a training needs assessment, followed by a job task analysis. Evaluation 
process was at different levels of the program design. The evaluation design used control groups. 
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and the primary measure was turnover rate, but absenteeism and overtime were also monitored. 
These measures were monitored before and after training (Phillips, 1994). 

Causeway Corporation implemented a Total Quality Management (TQM) training for 
white-collar workers. The whole training was related with strategic goals. Evaluation was part of 
the program design process, and tools to evaluate participants were part of the training content 
and used as guidelines for trainees. It was established clear accountability from participants, 
management, supervisors, and HRD staff. Participants estimated their contribution for ROI 
calculation. Results were both, in dollar and non-dollar terms (Phillips, 1994). 

An U.S. governmental institution applied the Instructional Systems Design (ISD) model 
with a team approach, and the four levels of evaluation. They applied pre and post course 
questionnaires to evaluate application level. T-test was used to analyze significance of changes in 
scores. Participants and managers estimated contribution by determining which competencies 
contributed to job success. These eight competencies where converted into monetary value. They 
determine the difference in the skills level before and after the course, and express this 
information in monetary value (Phillips, 1994). 

An insurance company had to change existing evaluation tools and techniques to make 
evaluation more comprehensive (four levels), reliable, and validate its tests. They compared 
results form three sources, namely trainees’ tests, survey, and performance appraisal scores. Also 
compared results from trainees who received the traditional and the new training program 
(Phillips, 1994). 

Magnavox Electronic Systems Company applied a training needs assessment; and 
according to skills gap, the solution would be literacy training program. The company found 
federal funding to finance the training program. Also, they established a partnership between 
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management steering committee and a public school. Evaluation was planned form the very 
beginning for four levels with state and federal representatives. They used pre- and post-tests to 
determine participants’ levels of reading and math skills. Key measures, namely productivity, 
scrap costs, and rework costs, were tracked for a year for training and control groups. Finally they 
reported direct and indirect costs (Phillips, 1994). 

Midwest Automotive plant compared forecasted financial benefits of structured and 
unstructured on the job training. The author reported costs for several stages, and they defined 
future performance value to determine benefits (Phillips, 1994). 

Arthur Andersen and Company developed a training program for tax consultants. SMEs 
validated training content. Evaluation design gathered data for levels 4 and 5, before and after 
training from both, training and control groups. Other intervening variables were identified to 
isolate effect of training, and they applied trend analysis (Phillips, 1994). 

Multi-Marques Inc. developed a needs analysis, and its results suggested the need to train 
supervisors to deal with employee problems and performance appraisal issues. Training program 
focused on supervisors’ experiences rather than new knowledge. Consequently, participants 
developed action plans to apply new skills on the Job (Phillips, 1994). 

The Coca-Cola bottling company of San Antonio determined the need of supervisory 
training. Participants developed action plans for eight modules to be reviewed and applied in 
coordination with their managers. This element was key for transferring new skills to the Job. 
Even though it is commonly considered a soft skill, supervisory training could be expressed in 
observable behavior to set specific goals (action plans). Time was given to design the program 
(one year), to implement it (eight months), and to see and measure results (Phillips, 1994). 
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International Oil Company implemented a needs analysis that determine a gap on 
customer service skills. Evaluation was part of design process and was implemented in four 
levels. Reported costs included design, implementation and evaluation phases. They determine 
unit measures, namely dealer complaints, pullout records (Phillips, 1994). 

At Midwest Banking Company, management pushed HRD department to demonstrate 
their contribution by calculating ROI. They chose Consumer Lending Seminar. Costs calculated 
and reported were for development, and implementation of the course. It was defined a unit of 
measurement that was tracked before and after training to measure improvement. The author 
identified external factors affecting increase in consumer loan volume, namely interest rates, 
special promotions, maturation of new employees job experience. By knowing these factors, they 
attempted to isolate training effect. Finally, intangible benefits were also identified (Phillips, 
1994). 

Information Services Inc. developed an interpersonal skills training. Authors stated the 
existence of confounding variables, namely training content in three locations with three different 
trainers; however, at the end of the training process they found no variance in results from 
different trainers. They gathered pre and post ratings of behavior, reported variable and fixed 
costs during design, development and implementation phases. To convert data into value, it was 
determined how much time does the person need to apply an specific category of skills. There 
were some indirect benefits not added to calculation of ROI (Phillips, 1994). 

Financial Services Inc. experienced an increased turnover rate of 63% for the position of 
branch manager trainee. The problem was outdated notions in the hiring process for branch 
manager trainee positions. Thus, consultants prepared cost projections for turnover problem and 
for the training program. It was developed a cause analysis for the problem and a job task 
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analysis. District sales managers were trained in recruiting strategies, interviewing guidelines, 
and evaluation guidelines. Besides, district sales managers were taught how to train new branch 
managers and were responsible of giving follow-up coaching. The evaluation design determined 
units of measure, namely number of terminations, terminations rate, and turnover rates. Data 
were tracked monthly during one year (Phillips, 1994). 

North County Electric & Gas Company implemented a training program on applied 
behavior management. CEO was involved in the process and participants knew they will be 
tracked and measured after training. As part of the contract, participants planned and 
implemented action plans. The training design used one-to-one coaching (participants and 
trainers) two hours per week. They also built and sustained a support system consisting on 
developing culture change, management commitment, steering group, employee communication 
and recognition, measuring and feedback systems. Finally, units of measurement were identified 
to determine monetary value of improvements (Phillips, 1994). 

Yellow Freight System developed a training needs assessment to develop a management- 
training program. Trainees created contracts that described how they would apply skills back on 
the job. Incentives and rewards were applied to participants, trainers, departments that support 
the effort, and managers. Participants estimated their contributions to calculate ROI (Phillips, 
1994). 

There are also, some research done on evaluation of training programs by calculating 
ROI. For example, Murray and Raffaele (1997) studied a Crosby quality-training intervention. 
The objective was to determine its long-term effects in terms of both quality level and dollar 
utility. Authors used a quasi-experimental design. The training process for managers, salaried 
employees, and hourly employees took a year. Measurements were taken during six years before 
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training (treatment), and follow-up measurements for the next five years after treatment. 
Researchers used an interrupted time-series design pooling six twelve-year series, and gathered 
data from four levels. The isolation of training effects included several strategies. One of them 
was statistical procedure, namely regression and definition of a “dummy” indicator. A second 
one was a detailed examination of annual reports to control no changes on environmental 
conditions that could affect results. Training costs included direct training costs, travel and 
reimbursement, consulting fees, facilities, time away from work, trainer time and fees, materials, 
and miscellaneous expenses. Managers emphasized that training was a major component of the 
intervention. 

Lyau and Pucel (1995) used a sample of 237 Taiwan firms to investigate labor- 
productivity returns from investments in training in a manufacturing industry. Subject Matter 
Experts validated the survey. Used of several procedures to assure return rate, namely a token of 
appreciation and follow-up telephone calls. The return rate was 68.4 percent, but only 55.3 
percent provided full cost data. The analysis was based on multiple regression that controlled 
other 14 variables that could affect the relationship between investment in training and labor 
productivity. 

Morrow, et al. (1997) implemented a four-year effort to investigate the effect and utility 
of managerial and sales/technical training. Authors applied a quasi-experimental design and a 
modified multi-attribute utility analysis model. Managerial training was found to have less effect 
and utility (mean ROI=45%) than sales/technical training (mean ROI=418%). 

McLinden, et al. (1993) assessed the return on investment of a training program on a tax 
consulting organization. Researchers determine pre-training trends, isolate the training effect 
from other factors. They found training had a substantive financial impact, namely pruticipants 
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Challenges on Calculating ROI and Cost-Benefit Analysis for Training Programs 

The training program for supervisors at Penske Truck leasing company applied control 
groups. However, results could not be generalized, because in real life variables can not be 
controlled like in experimental research (Phillips, 1994). 

Causeway Corporation, and the U.S. governmental institution only reported direct 
training costs. Likewise, Multi-marques Inc. calculated costs including only real disbursements 
of funds and on time invested in training by all categories of people (Phillips, 1994). 

An insurance company developed a training program that had some possible intervening 
variables, namely different instructors with dissimilar levels of expertise among them. However, 
there was no isolation of outside factors affecting performance on the job. Finally, training 
objectives were not validated by SMEs (Phillips, 1994). 

The training program implemented by Magnavox Electronic Systems Company had some 
barriers to guarantee participation. Thus, only half of primary targeted audience participated. 
Furthermore, 50 percent of participants did not finish the training. According with Gall, Borg, 
and Gall (1996) this is a threat to internal validity called subject attrition. At last, the organization 
expressed that the process for curriculum development was time consuming for both instructor 
and steering committee (Phillips, 1994). 

For some cases internal and external validity could be questioned. For example, results 
from Midwest Automotive Plant have a limited generalizability. Arthur Andersen and 
Company’s training program based their calculation with one group that was already working 
better. At Midwest Banking Company, the criterion for selecting the group that would be 
analyzed was availability. New requirements were added to participants, and new procedures 
were implemented to make sure that results would be favorable for the trainer. For example, the 




16 



Evaluating Training 16 



trainer asked to send the best prospects to this particular course. Costs calculated and reported 
were for development, and implementation of the course (Phillips, 1994). 

The Coca-Cola bottling company of San Antonio did not reported data on calculation of 
costs or isolation of training effect. Likewise, International Oil Company reported costs for 
design, implementation and evaluation phases. However, some important costs were not 
included, namely trainer’s salary and facilities’ costs (Phillips, 1994). 

Information Services Inc. developed an interpersonal skills training that had some 
confounding variables: training content in three locations with three different trainers. However, 
consultants reported no variance in results from different trainers (Phillips, 1994). 

At Financial Services Inc. a new approach of hiring process was implemented in addition 
with the training program. However, it was not implemented any method to isolate the effect of 
training form the new hiring strategy (Phillips, 1994). 

North County Electric & Gas Company implemented a training program that had a 
support system (culture change, management commitment, steering group, employee 
communication and recognition, measuring and feedback systems). However, training impact 
was not isolated from all those factors, and costs associated with the support system were not 
reported (Phillips, 1994). 

Yellow Freight System only reported general costs. They did isolate neither the impact of 
training nor the impact of management styles. Therefore, some political problems appear inside 
the organization, HRD and management were criticized of being taking credit of other 
department’s results. Therefore, CEO determined to stop collection of data for further analysis 
(Phillips, 1994). 
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Murray and Raffaele (1997) studied a Crosby quality-training intervention. Some 
limitations of the study are the use of percentages rather than daily or weekly rates that are more 
accurate measurements, observations were relatively few considering a twelve-year series, the 
evaluation lacked a control comparison, it has limited external generalizability because of the 
specific characteristics of the context in the organization. 

Lyau and Pucel (1995) used a sample of 237 Taiwan firms to investigate labor- 
productivity returns from investments in training in a manufacturing industry. They had a limited 
response rate, only 55.3 percent of companies returned completed data. Limited number of 
subjects in the analysis for the number of variables studied (131 subjects and 15 variables). 

Morrow, et. al. (1997) implemented a four-year effort to investigate the effect and utility 
of managerial and sales/technical training. They stated some limitations for the study, namely 
supervisors’ rates may be biased, threat to internal validity by diffusion of treatment, results may 
not generalize for other organizations. 

McLinden, et. al. (1993) assessed the return on investment of a training program on a tax 
consulting organization. They considered limitations such as resistance to adopt and apply the 
new knowledge and skills in the organization. 

After analyzing and discussing the cases, Phillips (1994) concluded the existence of 
weaknesses in ROI evaluations. However, Phillips would rather consider these weaknesses as 
future challenges for HRD professionals. These challenges are summarized in the following 
paragraphs. 

First, each organization applies different methods to determine costs. Sometimes one 
institution classifies an specific issue as a training cost, while other organization does not 
consider the same issue as a cost. Some authors even consider that a cost-benefit analysis should 




Evaluating Training 18 



consider the lost leisure of participants (Greenberg, 1997). Other authors stated that there is a 
lack of a universally accepted method to show the monetary benefits form training. Therefore, it 
is needed some standard cost data (Brauchle, & Koenecke, 1999; Phillips, 1994; Shelton, 1993). 

Second, some evaluation designs are not properly developed. Control groups are applied, 
but they do not reflect the real life. Surprisingly, some methods are not used, namely pre- and 
post- measuring, and time-series measurements (Phillips, 1994). 

Third, although most practitioners agree that several variables affect organizational 
outputs and performance (Phillips, 1994, 1997; Shelton, 1993; Torraco, 1995), many 
practitioners are not implementing appropriate methods to make an effort on isolating the 
influence training on organizational goals (Phillips, 1994). Some of these possible methods are 
trend analysis, forecasting, estimations, customer input, and subordinate input (Phillips, 1997, 
1996b). 

Fourth, data collection methods are often applied without pondering their advantages or 
disadvantages. The prevalent criterion is previous knowledge about the method rather than 
appropriateness. Therefore, there is a need of efforts to standardize these methods. 

Fifth, HRD practitioners are not paying enough attention to statistical techniques. Overall, 
statistics is being avoided even though can became an excellent source of power and support 
conclusions. For example, in some of the cases the sample size is too small; therefore, thus 
results can not be considered to have statistical support. This challenge corresponds with some 
other research efforts on criterion reliability, application of different evaluation designs, and 
tradeoffs between internal validity and statistical conclusion validity (Sacked, & Mullen, 1993; 
Yang, et. al., 1996). 
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Sixth, results from HRD programs are not converted into monetary values. This is an 
essential step for an ROI calculation. Even though the conversion process is somehow subjective, 
it is relevant compare benefits with costs (Phillips, 1994). Nonetheless, it is relevant to mention 
that Torraco (1995) considers that conclusions based on estimations are too subjective, thus, not 
reliable. 

Finally, Phillips (1997a) stated that HRD staff have to overcome some barriers to ROI 
implementation: a) the ROI process may add costs to the evaluation and may require more time 
invested; b) most of the time HRD staff members do not have the basic skills and understanding 
to apply the process; c) some HRD programs are still implemented for the wrong reasons, they 
are not based on an adequate needs assessment; therefore, some programs might not be beneficial 
for the organizations and will show a negative ROI; d) HRD staff members should be committed 
to implement the ROI process; otherwise, other pressures and priorities will not leave enough 
time for planning and assuring that the process is been implemented; e) some HRD departments 
are concerned about negative consequences of ROI results; and f) some HRD staff members have 
false assumptions regarding the ROI process. 

Conclusion 

After reviewing different authors, three major conclusions can be stated regarding what 
the future holds for evaluation of training programs through the ROI process. 

First, the implementation of the ROI process has relevant benefits for organizations: 

a) ROI is a way to measure the contribution from training and express it in monetary value; 

b) priorities can be made based on information of those programs that contribute more to the 
organization; c) ROI process helps to keep focused on results; and d) ROI process provides HRD 
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departments with valuable information to convince management that training is an investment 
and not an expense (Phillips, 1997a). 

Second, Brauchle and Koenecke (1999) state two important elements to be added in 
consideration, namely isolation and value of money over time. On one hand, the isolation process 
of the effects of training. Usually, training programs are not the only variable affecting 
performance improvement, namely work experience, economic changes, seasonal sales patterns, 
shifts in managerial styles, customer attitudes, and equipment breakdowns. Therefore, it is 
crucial to decrease the effect of other extraneous variables. On the other hand, ROI is not the 
final word. It is critical to calculate the present value of future dollars and the future value of 
present dollars. For example, the $100 dollars that a training program could help to save after 
three years have a real present value of $75. Besides, if an organization invests $100 dollars in 
training today, after three years, the future value of this amount will be $133. Therefore, the 
benefits from the training program must exceed $133. 

Third, according with Gall, Borg, and Gall, evaluation is a type of research. Some 
relevant issues like isolation of variables, internal and external validity, data collection methods, 
reliability of instruments, and use of statistical techniques are related with literature about 
research. Therefore, HRD practitioners would benefit from applying principles of research design 
to evaluation processes. 
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